Tactile emotion recognition provides a lot of valuable information in human-computer interaction, and it has strong application prospects in many aspects such as smart home and medical treatment. So this situation raises a question: How to quickly and efficiently let the robot perform the correct emotion recognition? In this work, we develop a lifelong learning algorithm which is based on the efficient dictionary learning technology, to tackle the tactile emotion recognition across different tasks. To verify the efficiency of the proposed (...) method, we applied it to two data sets for experimentation: Corpus of Social Touch (CoST) and our dataset(We built it with a 12X12 array sensor). The results show that the proposed lifelong learning algorithm achieves satisfactory results. (shrink)